SAS expander-side optimization for the re-discovery process

ABSTRACT

A simplification of the re-discovery process for initiators due to changes in the network is disclosed. If an initiator subscribes to change reports from a SAS expander, when that SAS expander detects a change in the network, it sends an SMP command back to the initiator, indicating the specific change in the network. Initiator BCN management and re-discovery of the entire network is therefore avoided.

FIELD OF THE INVENTION

This invention-relates to Serial Attached Small Computer SystemInterface (SAS) expanders, and more particularly to the isolation ofdevices connected to SAS expanders until verified, the binding of SerialAdvanced Technology Attachment (SATA) Tunneling Protocol (STP) SASaddresses to Serial Advanced Technology Attachment (SATA) devicesconnected to SAS expanders, and the optimizing of re-discovery when achange has occurred in the network.

BACKGROUND OF THE INVENTION

A SAS expander can be generally described as a switch that allowsinitiators and targets to communicate with each other, and allowsadditional initiators and targets to be added to the system. In SAS-1.1,the total number of initiators and targets was limited to on the orderof hundreds of devices. However, with SAS-2, SAS expanders can beconnected to each other up to 16 levels deep, and therefore thousands ofinitiators and targets may be connected.

FIG. 1 illustrates two SAS expanders 102 and 114 connected together inexemplary storage network 100. Each SAS expander 102 and 114 can includea plurality of Phy 104, expander connection manager (ECM) 106 forallowing pathways to be built between two Phy, expander connectionrouter (ECR) 108 which makes decisions regarding routing connectionsbetween Phy, and broadcast primitive processor (BPP) 110 for propagatingbroadcast change notifications (BCNs) to all other ports in the SASexpander except the port that caused the BCN to be generated.

FIG. 1 also shows an initiator 112 connected to a Phy 104 on first SASexpander 102, a SAS disk drive or other device 116 connected directly toanother Phy on the first SAS expander as a direct attached drive, andsecond SAS expander 114 connected to the first SAS expander. Newlyinserted device 116 is normally identified through the IDENTIFY process,and a broadcast change notification (BCN) is then generated by BPP 110in SAS expander 102 to notify other devices in storage network 100 ofthe change. External initiators can then use the SAS Management Protocol(SMP) Discover command to identify device attachments. In addition, ifdevice 116 was not properly seated in its connector, had a bad cable, orwas otherwise misbehaving in ways that produced errors, BPP 110 wouldnormally send out BCNs to other ports in SAS expander 102, which wouldcause all initiators (e.g. initiator 112) to perform a re-discoveryprocess for every device in the network to understand the contents ofthe SAS fabric. If device 112 was generating intermittent errors, manyBCNs (a “BCN storm”) could be transmitted, creating many re-discoveryprocesses to be performed. Alternatively, if a downstream SAS expanderhad an illegal configuration such as two Phy (on the same expander)connected together, this could also cause a BCN storm. Constant BCNswill impact the usability of a SAS network.

Therefore, there is a need to be able to isolate, test and validatedevices before they are made visible to the network.

Two device types, SAS and Serial Advanced Technology Attachment (SATA),can commonly be connected to a storage network using SAS expanders. SASdevices have a unique 64-bit SAS address already assigned to them.However, current SATA devices do not have a SAS address. This isimportant, because to enable a SAS expander-attached SATA device to bevisible to other devices in the network, the Phy to which it is attachedmust be assigned a unique SAS address. (Note that every Phy in a SASdomain must have a unique SAS address (with the exception of wide-ports,which all connect to the same device and can therefore share a commonSAS Address)). Because SATA devices do not present a SAS address, theyare assigned an address by the SAS expander. Each expander portmaintains an SATA Tunneling Protocol (STP) SAS address, which identifiesa SATA device connected to the port. However, if the SATA device isremoved and replaced by a new SATA device, the old STP SAS addressremains bound to the port, and thus any outstanding input/output (I/O)requests targeted to the removed SATA device will instead be deliveredto the new SATA device connected to the same port, potentiallycorrupting the new SATA device.

FIG. 1 illustrates exemplary SAS expander 114 connected to SAS and SATAdevices 118 and 120, respectively, in storage network 100, and theaddressing problem created by attached SATA devices. SAS device 118connected to Phy 122 within SAS expander 114 has its own SAS address Z,and thus if the SAS device is moved at 132 to another Phy (see Phy 134),the SAS address follows the SAS device. In contrast, SATA device 120 isconnected to STP port 124 and Phy 126. STP port 124 provides translationfunctionality between SATA and SAS. SATA device 120 is assigned anaddress X which is bound to Phy 126, so that if the SATA device is movedto another port at 136 and a new SATA device is inserted in its place,the address X stays with the Phy, and the new SATA device receives theold STP address X. When this happens, outstanding I/O requests maycomplete to the new SATA device plugged into Phy 126 with address X, andthe data will be written to or read from the wrong device, resulting incorruption.

One initiator-based solution to this problem is as follows. When a SATAdevice is removed, a BCN is generated. Because the source of the BCNcannot be distinctly identified down to an expander and port, all SATAdrives in the SAS domain are placed in a hold state. All existing SATAI/O requests are aborted and new SATA I/O requests are rejected untilthe driver re-validates the SATA devices and removes them from the holdstate.

A disadvantage of this solution is that any change in the domain(indicated by a BCN) causes all SATA devices to be placed into the holdstate (as there is no way to know what changed). Additionally, any BCN(such as a SAS hot-insert) causes all SATA devices to be placed into thehold state. While this provides the maximum protection, it also placesan additional burden on initiators for managing domain changes (aboveand beyond normal discovery).

Therefore, there is also a need to provide some level of persistentbinding for a SATA device. If a SATA device is moved from one port toanother, the STP SAS address should follow it to the new port.

As mentioned above, in earlier versions of SAS (e.g. SAS-1.1), onlyseveral hundred devices could practically be attached in the network.However, with SAS-2, due to changes such as higher link rates (3 G to 6G) and innovations such as connection multiplexing which allows a singlelink to be time-division multiplexed to improve access, more than athousand devices can be attached. This large number of devices can causeproblems, as will be discussed below.

FIG. 1 illustrates initiator 112 and several SAS expanders 102 and 114chained together. Each SAS expander 102 and 114 is self-configuring—thatis, each expander takes care of its own route-table programming. Eachexpander has a plurality of ports to which devices or other expandersmay be attached.

SMP allows initiators to perform discovery, in which each initiatorcommunicates with every device to discover what is attached within thenetwork. Each SAS expander may have to send an SMP REPORT_GENERALcommand to each initiator, indicating how many devices are attached andproviding other basic SAS expander information back to the initiator.Additionally, one SMP DISCOVER command would have to be sent out foreach Phy on each SAS expander, and SAS expanders typically have 36-38Phys. Initiators must therefore keep track of the device tree (whichdevices are connected to which expander, etc.).

Similarly, if a device is unplugged or otherwise changed, the discoveryprocess requires that the SAS expander to which the device was attachedsend a BCN is sent out to all devices in the network. The BCN provides anotification of fabric changes without specific details. The initiatormust then determine which devices are now unavailable (the expander, andall devices behind it). To accomplish this, each initiator must performa full re-discovery of the entire SAS domain. Full rediscovery involvesmany SMP DISCOVER commands to pinpoint one or two changes that haveoccurred in the fabric. For large SAS networks of 1000+ drives(requiring as many as 42 expanders), thousands of SMP commands can beperformed to identify what changed in the network. In other words, thereis no mechanism available to determine what changed in the SAS networkwithout enumerating every Phy of every SAS expander. This process can betime-consuming, and in multi-initiator settings, can be performed bymultiple initiators simultaneously. This has the effect of burdening theSAS network with management traffic and reducing the available I/Obandwidth. Thus, SAS does not scale well to large networks.

Therefore, there is also a need to enable an initiator to quickly andefficiently discover SAS network by obtaining re-discovery informationabout all devices in a network without having to perform a fullre-discovery process.

SUMMARY OF THE INVENTION

Embodiments of the invention are directed to the use of existing SASmechanisms to isolate new devices until they can be validated ashealthy. When an internal PHY_ENABLE event is received from the Phy, theSAS expander will hold off on sending a BCN, and perform a conventionalnegotiation with the attached device to determine what kind of device isconnected to the Phy (through the IDENTIFY sequence). The device canthen be interrogated further to determine if it is inserted and workingproperly. Once the device is validated (via a port-test-before-insertionapproach) and found to be healthy, the fabric is notified via a BCN, anda management entity can move the device from an isolated state into avisible state in the SAS domain. If the device was in a state thatcauses constant BCNs to be generated, the SAS expander can notify anexternal management device and then disable the device (either bypowering the drive down, or by disabling its Phy). This protects therest of the SAS network from misbehaving devices that could otherwisegenerate a BCN storm, and makes the network more manageable.

One advantage of this approach is that while the device is isolated, itcan do no harm to the rest of the SAS network. The only entity that isable to communicate with the device in this condition is the SASexpander itself (the SMP initiator virtual phy and SSP/SATA initiatorvirtual phy within the SAS expander), which will perform verificationtechniques similar to the port-test-before-insertion (PTBI) techniquesfor Fibre Channel (FC) switches previously described in U.S. PatentPublished Application No. 2006/0146698, the contents of which areincorporated by reference herein. The SAS expander may also disable thedrive if it is known to present problems (e.g. a drive for which thereare known incompatibilities).

Embodiments of the invention are also directed to providing persistentbinding of STP SAS addresses to SATA devices so that they can be movedto different insertion points (ports) within a SAS expander and stillproperly receive I/O requests. When a SATA device is inserted into theSAS expander, it is placed into a provisional state while it isinterrogated (using PTBI-like communications and a temporary address) tovalidate and obtain information about the attached device. Thisinformation may represent many different fields, such as manufacturername, serial number, etc. Certain fields may be combined using a hashingfunction to obtain a unique ID for the SATA device.

After the unique ID has been generated for the SATA device, a search forthe unique ID within a table can be performed, and if the unique ID isfound in the table, the STP SAS address associated with the unique IDcan be assigned to that Phy. In this manner, the same STP SAS addresswill be assigned to the Phy connected to a particular SATA device,regardless of where the device is connected to the SAS expander. If theunique ID is not found in the table, a new 64-bit STP SAS address can beassigned to that unique ID, and a new table entry can be created.

Embodiments of the invention are also directed to simplifying there-discovery process for initiators by accepting a unicast-basednotification which communicates the specifics of the change. Inparticular, an SMP command is sent from a SAS expander to an initiator,indicating the specific change in the network. Initiator BCN managementis therefore avoided. Because the SAS expander knows exactly whatchanged, the expander simply notifies each initiator of the specificchange, eliminating the need for each initiator to search the networkfor the change. This reduces the thousand or more SMP transactions perinitiator to one SMP transaction per initiator (per change). In otherwords, the SMP DISCOVER data is pushed directly to initiators from theaffected expander, requiring no re-discovery to be performed (andminimizing the SMP thrashing that can occur).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates two SAS expanders connected together in an exemplarystorage network.

FIG. 2 illustrates an exemplary initiator connected to a storage networkand employing the use of masking (making a Phy's attachment invisible toan initiator) to isolate devices prior to their verification andinsertion into a network according to embodiments of the invention.

FIG. 3 is a ladder diagram illustrating the exemplary communicationsbetween the SAS expander firmware and other logic and functions withinthe SAS expander according to embodiments of the invention.

FIG. 4 illustrates an exemplary SAS expander utilizing a table formapping device information to a SAS address according to embodiments ofthe invention.

FIG. 5 is an exemplary ladder diagram illustrating communicationsbetween SAS expanders and initiators to provide an initiator withinformation about a change in the network according to embodiments ofthe invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In the following description of preferred embodiments, reference is madeto the accompanying drawings which form a part hereof, and in which itis shown by way of illustration specific embodiments in which theinvention may be practiced. It is to be understood that otherembodiments may be utilized and structural changes may be made withoutdeparting from the scope of the preferred embodiments of the presentinvention.

FIG. 2 illustrates exemplary SAS expander 202 connected within storagenetwork 200 and employing the isolation of devices prior to theirverification and insertion into a network according to embodiments ofthe invention. In FIG. 2, SAS expander 202 is shown with processor 204,flash 206, virtual Phy 208, and switch complex 210 including an ECM, ECRand BPP. Optionally connected to SAS expander 202 is SMP initiator 212,which contains zoning manager 214 that is responsible for performingzoning functions. Multiple ports 216 are present on SAS expander 202,each port containing a separate Phy 218. A device 220 may be connectedto each Phy 218.

In a conventional SAS expander, if it is determined that a device isconnected to a Phy, a negotiation or validation process is commencedbetween the SAS expander and the device to determine information aboutthe device and verify that the device was connected and workingproperly. In either case (in both the nominal and exception (error)case), once the Phy changes status as a result of the connected device,the SAS expander sends out a BCN to all other devices in the network,which causes all initiators to perform a rediscovery process torediscover all devices in the network. One exception to transmission ofa BCN is when a new device is inserted but does not pass the PTBIprocess. In this case, no BCN is sent because no change occurred in theSAS domain.

However, according to embodiments of the invention, when an PHY_ENABLEevent is received through virtual Phy 208, indicating that a device hasbeen connected to a regular Phy, SAS expander 202 will perform aconventional negotiation with the device (speed determination, IDENTIFYexchange, etc.) but will hold off on the generation of a BCN while theSAS expander validates the attached device. During this time, if aninitiator performed discovery, the Phy would return a SMP PHY_VACANTresponse, indicating that there is no device attached to the Phy (whenin fact there is). Thus, the device remains isolated and invisible toexternal initiators. With the IDENTIFY frames exchanged, the SASexpander knows which kind of device is behind the Phy and caninterrogate it further (either through an SMP and Serial SCSI Protocol(SSP) dialogue, or through an SMP and SATA dialogue). Each Phy 218 forwhich a connected device has been detected will then perform a port testbefore insertion (PTBI)-like validation process using SMP and SCSIcommands to ensure that the device is valid and that it is of a typesupported by SAS expander 202. For example, Phy 218, which is isolatedand is connected to device 220, will perform the validation process onthe device.

If it is determined that device 220 is bad, the Phy can be disabled orthe device itself can be powered down so that no commands can be sent todevice 220, and a notification can be sent to a system administrator oran external management device. Note that no BCN is sent out, and thusthe bad device does not cause all initiators to perform a re-discoveryprocess. This protects the rest of the SAS network from misbehavingdevices that could otherwise generate a BCN storm, and makes the networkmore manageable.

On the other hand, if device 220 is determined to be good, firmware inSAS expander 202 then exposes the Phy and its attached device to thelarger SAS fabric (i.e. the Phy no longer emits an SMP PHY_VACANTresponse for SMP DISCOVER requests). Having been exposed to the SASfabric, all other devices in the network, including zoning manager 214,will now be able to “see” and communicate with device 220.

FIG. 3 is a ladder diagram 300 illustrating the exemplary communicationsbetween SAS expander firmware 302 and other logic and functions 304within the SAS expander according to embodiments of the invention. Inthe first communication 306, each Phy is initialized and isolated. ThePhy are also enabled so that they can negotiate with a Phy to which theyare attached and exchange information about the Phy. Next, each Phy isinterrogated at 308 using an IDENTIFY exchange to validate and obtaininformation about the device attached to the Phy, and theself-configuring bit is reset. These aforementioned steps are performedat startup or boot time of the SAS expander, or whenever a new device isattached to the SAS expander.

At some later time, if a SAS device is hot-inserted to the SAS expander,a PHY_ENABLE is sent to SAS expander firmware 302, indicating a Phychange. In response to the Phy change, if the inserted device is a SASdevice (as determined by the IDENTIFY exchange), an SMP/SSP Initiatorwithin the SAS expander interrogates the attached devices at 312 togather information about the attached SAS device. No BCN is sent out atthis time. During this time, if an initiator performed discovery, thePhy would return a SMP PHY_VACANT response, indicating that there is nodevice attached to the Phy (when in fact there is). Thus, the deviceremains isolated and invisible to external initiators. The SMP/SSPinitiator determines the health of the device, and then either leavesthe Phy isolated if the device is bad, or reveals the Phy to the largerSAS fabric (i.e. the Phy no longer emits an SMP PHY_VACANT response forSMP DISCOVER requests). Potential tests for a SAS drive include, but arenot limited to, SMP Phy Test Function, SCSI Test Unit Ready (spin-uptest), SCSI Inquiry (drive compatibility test), SCSI Request Sense(obtain error data), and SCSI Send Diagnostic Command (self-test). TheSMP/SSP initiator then sends a zoned-broadcast message to zoning manager214 so that the zoning manager can further configure the Phy to theappropriate zone group.

If the inserted device is a SATA device (as determined by the Signatureframe (similar to the IDENTIFY exchange)), an SMP/SATA initiator withinthe SAS expander interrogates the drive at 314 to gather informationabout the device. The SMP/SATA initiator at the virtual phy determinesthe health of the device, and then either leaves the device as isolatedif the device is bad, or makes the device visible in the SAS domain andemits a BCN. Potential tests for a SATA drive include, but are notlimited to, SMP Phy Test Function, ATA Identify Device, ATA Check PowerMode, ATA Get Media Status, and ATA Smart Execute Off-Line Immediate.The SMP/SATA initiator then sends a zoned-broadcast message to zoningmanager 214 so that the zoning manager can further configure the phy tothe appropriate zone group.

In an alternative embodiment, if supported, isolation can beaccomplished using zone groups. Some versions of SAS employ the conceptof assigning zone groups to devices such that only those devices withcompatible zone groups can communicate with each other. In SAS, eightbits are used to define zone groups. Zone groups 0-127 are valid zonegroups, while zone groups 128-255 are reserved zone groups.

According to this alternative embodiment, the SAS expander, under thecontrol of firmware executed by a processor within the SAS expander,assigns a unique reserved zone group number to each Phy in the SASexpander. By assigning a unique reserved zone group number to each Phy,no Phy (except the virtual Phy within the SAS expander) can see anyother Phy. When a Phy_enable event is received through the virtual Phy,indicating that a device has been connected to a regular Phy, the SASexpander will perform a conventional negotiation with the device (speeddetermination, IDENTIFY exchange, etc.). With the IDENTIFY framesexchanged, the SAS expander knows which kind of device is behind the Phyand can interrogate it further (either through an SMP and SSP dialogue,or through an SMP and SATA dialogue). Each Phy for which a connecteddevice has been detected will then perform a PTBI-like validationprocess using SMP and SCSI commands to ensure that the device is validand that it is of a type supported by the SAS expander.

If a Phy determines that its attached device is bad, the Phy can bedisabled or the device itself can be powered down so that no commandscan be sent to the device, and a notification can be sent to a systemadministrator or an external management device. Note that no BCN is sentout, and thus the bad device does not cause all initiators to perform are-discovery process. This protects the rest of the SAS network frommisbehaving devices that could otherwise generate a BCN storm, and makesthe network more manageable.

On the other hand, if the attached device is determined to be good,firmware in the SAS expander then changes the Phy to have a unique validzone group number selected from a subset of the valid zone group numbers0-127. For example, the subset of valid zone group numbers can be 0-3.In addition, the SAS expander notifies the rest of the fabric bytransmitting a BCN. Having been assigned a unique valid zone groupnumber selected from the subset of valid zone group numbers, all otherdevices connected to Phy with valid zone types in the network, includingthe zoning manager, will now be able to “see” and communicate with thedevice. When the zoning manager sees that the Phy now has a valid zonegroup number, it can then re-assign that Phy to have a unique valid zonegroup number outside the subset of valid zone group numbers.

In a further alternative embodiment, instead of using a zoning manager,flash memory within SAS expander may contain a mapping of deviceaddress, slot or Phy to zone group number. Thus, when a device isdetermined to be good, the firmware in the SAS expander can change thePhy to have a valid zone based on the address, slot or Phy of the device(which was determined during the validation process) and the mappingstored in the flash memory.

FIG. 4 illustrates an exemplary SAS expander 402 in a storage system 400utilizing a table 404 for mapping a device ID 406 to an STP SAS address408 according to embodiments of the invention. In the example of FIG. 4,if SATA device 410 is connected to SAS expander 402 at an STP port 412,instead of binding the STP SAS address X to Phy 414 within STP port 412,table 404 is utilized. Table 404 contains a mapping of a unique ID 406to a 64-bit STP SAS address 408. The STP SAS addresses 408 are not boundto any particular port.

When a SATA device is inserted into the SAS expander, it is placed intoa provisional (isolated) state while it is interrogated (using PTBI-likecommunications and a temporary address). As mentioned above, during thevalidation process, each Phy (and its attached device) are interrogatedusing an IDENTIFY exchange to validate and obtain information about thePhy and the attached device. This information may represent manydifferent fields, such as manufacturer name, serial number, etc. One ofthese fields could be the World Wide Name (WWN) field (e.g., 4-16bytes). The WWN is assigned by the IEEE and must be used by themanufacturer of the SATA device. Certain fields may be combined using ahashing function to obtain a unique ID for the device.

Thus, in the example of FIG. 4, after a unique ID has been generated forSATA device 410, under the control of a processor and/or logic withinSAS expander 402, a search for the unique ID within table 404 can beperformed, and if the unique ID is found in the table, the 64-bit STPSAS address associated with the unique ID can be assigned to that Phy.If the unique ID is not found in the table, a new 64-bit STP SAS addresscan be assigned to that unique ID, and a new table entry can be created.In an alternative embodiment, the new 64-bit STP SAS address may beselected from previously used addresses using a recursive or LeastRecently Used (LRU) algorithm. In other words, the available STP SASaddresses (for STP port assignment) are sorted in last used order to puttemporal distance between STP SAS address usages. After an STP SASaddress is bound to the port, the port is migrated from the provisionalstate to the operational state and a BCN generated to indicate domainchange.

The same STP SAS address will thereafter be assigned to the Phyconnected to SATA device 410, regardless of where the SATA device isconnected to the SAS expander. Thus, even if SATA device 410 is moved at422 to STP port 424, the same process described above is performed,unique ID A is once again generated at STP port 424, and the table 404is once again used to assign STP SAS address “X” to Phy 426.

It should be noted, however, that if SATA device 410 is removed andconnected to a different SAS expander 416 (see relocation 420), that newSAS expander will only have access to its own table 418, and thus a newSTP SAS address would be assigned to that SATA device the first time itis connected to the new SAS expander, using table 418. Any outstandingI/O requests for the old STP SAS address would not complete, and wouldbe rejected.

Embodiments of the invention described above therefore provide a SASexpander-based solution to the problem of assigning STP SAS addresses toSATA drives that affords maximum protection to SATA devices in a verynon-disruptive way. The SAS expander-based solution both solves theproblem locally, and scales well into larger SAS fabrics.

Another embodiment of the invention enables a SAS expander to provide anindication to an initiator of what changes occurred in the network,rather than require each initiator to rediscover all devices. Inparticular, when a change is detected in the network, an SMP command issent from the SAS expander to the initiator, indicating the specificchange in the network. Because the SAS expander knows exactly whatchanged, rather than relying on each initiator to perform BCN managementto search the network for the change, the SAS expander simply notifieseach initiator of the specific change. This reduces the thousand or moreSMP transactions (in a large SAS fabric) per initiator to one SMPtransaction per initiator (per change).

FIG. 5 is an exemplary ladder diagram illustrating communicationsbetween SAS expanders 500 and 504 and initiators 502 and 506, where SASexpanders 500 and 504 and initiator 502 are enhanced according toembodiments of the invention, and initiator 506 is conventional indesign. Both initiators 502 and 506 are connected to a single SASdomain. The firmware in the initiators must be able to receive andrespond to the new SMP commands discussed below.

In the initial discovery phase, both initiators behave in the same way(full enumeration of the domain). Expander 500 sends a BCN 508 toinitiator 502 (and all other endpoints), and the initiator performs afull topology discovery at 510, discovering both expander A and B (see512 and 514). Similarly, expander 504 sends a BCN 516 to initiator 506,and the initiator performs a full topology discovery at 518, discoveringboth expander A and B (see 522 and 520). During this discovery process,each SAS expander 500 and 504 sends an SMP REPORT_MANU_INFO command toinitiators 502 and 506, respectively, to provide attached devicemanufacturer information to the initiators. The SAS expanders 500 and504 also send another command to initiators 502 and 506, respectively,to indicate the SAS expanders' capabilities. Once an initiator completesthe initial discovery process, it has a baseline of the entire SASfabric (all devices that exist at that epoch in time). At this point,the fabric is stable.

Initiator 502, under the control of firmware executed by a processorwithin the initiator according to embodiments of the invention, andbeing aware of the enhanced capabilities of SAS expanders 500 and 504according to embodiments of the invention, then contacts each enhancedSAS expander (SMP Target) and subscribes to change reports by sendingout an SMP CONFIG_GENERAL command (an extension of the conventional SMPCONFIG_GENERAL command) at 524 and 526 to each SAS expander. Note thatwhen initiator 502 performs an SMP REPORT_MANU_INFO command, it knowsthat the expander is a SAS expander according to embodiments of theinvention. With this, the initiator performs an SMP REPORT_CAPABILITIEScommand to know what capabilities the SAS expander has.

Because initiator 502 subscribed to change reports, when a new device isinserted into SAS expander 504, BCNs transmitted by the SAS expanderwill be ignored by the initiator. Instead, an SMP REPORT_CHANGE command528, which is an SMP transaction and a variation on a conventional SMPDISCOVER command and an SMP REPORT_PHY_SATA command, is unicast directlyfrom SAS expander 504 to subscribing initiator 502. The SMPREPORT_CHANGE command (vendor extension) 528 notifies initiator 502 thata new device has been inserted into SAS expander 504, and providesinformation about that new device. In particular, the SMP REPORT_CHANGEcommand 528 provides SMP DISCOVER command data (essentially the samedata that is returned in a response to an SMP DISCOVER command,indicating, for a given Phy on the SAS expander, what device isattached), plus other vendor-unique information. The SMP REPORT_CHANGEcommand 528 requires that initiator 502 support an SMP Target. Afterinitiator 502 receives the SMP REPORT_CHANGE command 528 and itsassociated data, the initiator returns an SMP REPORT_CHANGE_RESPONSE 530to expander 504. Nothing is provided back to the SAS expander, otherthan the initiator confirms receipt of the command by setting theFunction Result within the SMP response to SUCCESS. The result is anoptimized rediscovery of the SAS fabric that allows a SAS expander tocommunicate in a single transaction what device change occurred. An SMPinitiator in the SAS expander 504 performs this operation for eachexternal initiator that has subscribed for change reports, freeing eachsubscribing initiator from having to rediscover the entire network.

However, if a conventional SAS expander notes a change in the networkand sends out a BCN, an enhanced initiator according to embodiments ofthe present invention will temporarily ignore the BCN and wait for anSMP REPORT_CHANGE from that SAS expander. Receiving none after apredetermined period of time, the enhanced initiator will rediscover theentire network in the conventional manner. In an alternative embodiment,because the enhanced initiator knows that a BCN could only have beensent from a conventional SAS expander, the initiator will onlyrediscover the conventional SAS expanders. Also, when expanders areadded or removed, a new fabric baseline must be performed (as shown inFIG. 5).

Referring again to the example of FIG. 5, because initiator 506 isconventional, it does not send out any SMP CONFIG_GENERAL command to anySAS expanders to subscribe to change reports. Because initiator 506 didnot subscribe to change reports, when a new device is inserted intoexpander 504 and BCN 532 is transmitted to initiator 506 to provide anotification of a fabric change, initiator 506 performs a fullre-discovery 534 (which is essentially the same as the initial discoveryprocess) in the conventional manner to identify the change thatoccurred. Embodiments of the invention can therefore be consideredSAS-friendly because the behavior of an initiator will be unaffected (itwill perform SAS re-discovery in the conventional manner, for which SASexpander support remains) if the initiator is not aware of the enhancedcapability of the SAS expander and fails to subscribe to change reports.

Embodiments of the invention also support existing SAS behaviors such asSATA device insertion. If the initiator must spin up a SATA device, anSMP REPORT_GENERAL command will indicate that the SATA device is in aspinup-hold state (via the negotiated physical link rate (NPLR) in theSMP DISCOVER data). The initiator must then issue an SMP PHY_CONTROLcommand to transition the SATA device out of the spinup-hold state.Commonly, a new BCN will result (indicating the drive has spun-up), butin the case of initiator 502, a new SMP REPORT_CHANGE command 528 willresult, with an updated NPLR.

As described above, embodiments of the invention can advantageouslyprovide a large reduction in SAS management traffic for an initiator,particularly in large SAS networks, a large reduction in SAS managementtraffic in the interior of the SAS network (particular inmulti-initiator environments), a SAS protocol friendly method capable ofoperating alongside standard SAS-1.1 or SAS-2 devices, betterperformance of SAS networks in device change scenarios, and the abilityfor SAS to support very large networks of drives without the bandwidthreduction due to rediscovery.

Although the present invention has been fully described in connectionwith embodiments thereof with reference to the accompanying drawings, itis to be noted that various changes and modifications will becomeapparent to those skilled in the art. Such changes and modifications areto be understood as being included within the scope of the presentinvention as defined by the appended claims.

1. A system for efficient initiator re-discovery of devices connected ina storage network when a change in the storage network has occurred,comprising: a Serial Attached SCSI (SAS) expander configured forreceiving a subscription to change reports from one or more initiators,and when a change in the storage network is detected by the SASexpander, sending a modified change report to the one or more initiatorsfrom which a subscription to change reports was received, the modifiedchange report containing data from a response to a DISCOVER commanddefining a device change.
 2. The system of claim 1, the subscription tochange reports comprising an SMP_config_general_extended command.
 3. Thesystem of claim 1, the modified change report comprising anSMP_report_change command.
 4. The system of claim 1, further comprising:an initiator configured for sending the subscription to change reportsto the SAS expander, and receiving the modified change report from theSAS expander when the change in the storage network is detected by theSAS expander.
 5. The system of claim 4, the initiator further configuredfor: receiving a broadcast change notification (BCN) from the SASexpander; taking no action as a result of the received BCN for apredetermined period of time since the receipt of the BCN; and if nomodified change report is received from the SAS expander by the end ofthe predetermined period of time, performing re-discovery of thenetwork.
 6. The system of claim 5, the initiator further configured forperforming re-discovery of only those SAS expanders to which nosubscription to change reports was sent.
 7. The system of claim 1,further comprising one or more initiators and one or more devicescoupled to the SAS expander.
 8. A storage network comprising the systemof claim
 7. 9. An initiator for re-discovering devices connected in astorage network when a change in the storage network has occurred,comprising: a processor programmed for sending a subscription to changereports to a SAS expander, and receiving a modified change report fromthe SAS expander when the change in the storage network is detected bythe SAS expander, the modified change report containing data from aresponse to a DISCOVER command defining a device change.
 10. Theinitiator of claim 9, the subscription to change reports comprising anSMP_config_general_extended command.
 11. The initiator of claim 9, themodified change report comprising an SMP_report_change command.
 12. Theinitiator of claim 9, the processor further programmed for: receiving abroadcast change notification (BCN) from the SAS expander; taking noaction as a result of the received BCN for a predetermined period oftime since the receipt of the BCN; and if no modified change report isreceived from the SAS expander by the end of the predetermined period oftime, performing re-discovery of the network.
 13. The system of claim12, the initiator further configured for performing re-discovery of onlythose SAS expanders to which no subscription to change reports was sent.14. A method for efficient initiator re-discovery of devices connectedin a storage network when a change in the storage network has occurred,comprising: receiving a subscription to change reports from an initiatorat a Serial Attached SCSI (SAS) expander, and when a change in thestorage network is detected by the SAS expander, sending a modifiedchange report to the initiator from which a subscription to changereports was received, the modified change report containing data from aresponse to a DISCOVER command defining a device change.
 15. The methodof claim 14, the subscription to change reports comprising anSMP_config_general_extended command.
 16. The method of claim 14, themodified change report comprising an SMP_report_change command.
 17. Themethod of claim 14, further comprising: sending the subscription tochange reports to the SAS expander from the initiator, and receiving themodified change report from the SAS expander at the initiator when thechange in the storage network is detected by the SAS expander.
 18. Themethod of claim 17, further comprising: receiving a broadcast changenotification (BCN) from the SAS expander at the initiator; taking noaction at the initiator as a result of the received BCN for apredetermined period of time since the receipt of the BCN; and if nomodified change report is received from the SAS expander at theinitiator by the end of the predetermined period of time, performingre-discovery of the network at the initiator.
 19. The method of claim18, further comprising performing re-discovery at the initiator of onlythose SAS expanders to which no subscription to change reports was sent.20. A method for re-discovering devices connected in a storage networkwhen a change in the storage network has occurred, comprising: sending asubscription to change reports to a SAS expander from an initiator, andreceiving a modified change report from the SAS expander at theinitiator when the change in the storage network is detected by the SASexpander, the modified change report containing data from a response toa DISCOVER command defining a device change.
 21. The method of claim 20,the subscription to change reports comprising anSMP_config_general_extended command.
 22. The method of claim 20, themodified change report comprising an SMP_report_change command.
 23. Themethod of claim 20, further comprising: receiving a broadcast changenotification (BCN) from the SAS expander at the initiator; taking noaction at the initiator as a result of the received BCN for apredetermined period of time since the receipt of the BCN; and if nomodified change report is received from the SAS expander at theinitiator by the end of the predetermined period of time, performingre-discovery of the network at the initiator.
 24. The method of claim23, further comprising performing re-discovery at the initiator of onlythose SAS expanders to which no subscription to change reports was sent.25. A computer-readable medium comprising program code for efficientinitiator re-discovery of devices connected in a storage network when achange in the storage network has occurred, the program code for causingperformance of a method comprising: receiving a subscription to changereports from an initiator at a Serial Attached SCSI (SAS) expander, andwhen a change in the storage network is detected by the SAS expander,sending a modified change report to the initiator from which asubscription to change reports was received, the modified change reportcontaining data from a response to a DISCOVER command defining a devicechange.
 26. The computer-readable medium of claim 25, the subscriptionto change reports comprising an SMP_config_general_extended command. 27.The computer-readable medium of claim 25, the modified change reportcomprising an SMP_report_change command.
 28. The computer-readablemedium of claim 25, the program code further for causing performance ofa method comprising: sending the subscription to change reports to theSAS expander from the initiator, and receiving the modified changereport from the SAS expander at the initiator when the change in thestorage network is detected by the SAS expander.
 29. Thecomputer-readable medium of claim 28, the program code further forcausing performance of a method comprising: receiving a broadcast changenotification (BCN) from the SAS expander at the initiator; taking noaction at the initiator as a result of the received BCN for apredetermined period of time since the receipt of the BCN; and if nomodified change report is received from the SAS expander at theinitiator by the end of the predetermined period of time, performingre-discovery of the network at the initiator.
 30. The computer-readablemedium of claim 28, the program code further for causing performance ofa method comprising performing re-discovery at the initiator of onlythose SAS expanders to which no subscription to change reports was sent.31. A computer-readable medium comprising program code forre-discovering devices connected in a storage network when a change inthe storage network has occurred, the program code for causingperformance of a method comprising: sending a subscription to changereports to a SAS expander from an initiator, and receiving a modifiedchange report from the SAS expander at the initiator when the change inthe storage network is detected by the SAS expander, the modified changereport containing data from a response to a DISCOVER command defining adevice change.
 32. The computer-readable medium of claim 31, thesubscription to change reports comprising an SMP_config_general_extendedcommand.
 33. The computer-readable medium of claim 31, the modifiedchange report comprising an SMP_report_change command.
 34. Thecomputer-readable medium of claim 31, the program code further forcausing performance of a method comprising: receiving a broadcast changenotification (BCN) from the SAS expander at the initiator; taking noaction at the initiator as a result of the received BCN for apredetermined period of time since the receipt of the BCN; and if nomodified change report is received from the SAS expander at theinitiator by the end of the predetermined period of time, performingre-discovery of the network at the initiator.
 35. The computer-readablemedium of claim 34, the program code further for causing performance ofa method comprising performing re-discovery at the initiator of onlythose SAS expanders to which no subscription to change reports was sent.